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Abstract 

An  explicit  theory  for  “measures  of  effectiveness”  (MOEs  as  they  are  commonly  referred 
to)  does  not  exist.  As  a  result  several  definitions  for  MOEs  have  been  advanced,  that 
while  similar,  do  not  provide  the  needed  insight  in  to  system  performance  evaluation. 

Original  studies  performed  by  the  Military  Operations  Research  Society’s  (MORS) 
Command  and  Control  workshop  in  the  mid-  1980s  laid  a  foundation  for  a  more 
theoretical  approach  that  was  well  received  within  the  MORS  community.  However, 
little  has  been  done  in  the  last  decade  to  further  this  work.  Most  papers  in  the  latter  part 
of  the  1990s  make  no  reference  to  the  workshop  reports  and  other  published  papers  that 
resulted  from  the  workshop. 

This  paper  will  present  a  review  of  the  original  work  performed  by  MORS  as  well  as 
relevant  material  that  has  been  published  in  the  intervening  years.  It  will  extend  the 
original  body  of  work  using  the  systems-of-systems  perspective  originally  developed  by 
Russell  Ackoff.  A  concise  systems  based  definition  of  MOEs  will  be  derived  using  this 
framework.  The  paper  will  also  present  the  framework  for  a  consistent  mathematical 
theory  for  MOEs. 

Introduction 

The  earliest  references  to  MOEs  are  found  in  the  post  WWII  report  of  the  Operations 
Evaluation  Group  subsequently  published  as  “Methods  of  Operations  Research”  [25]. 
Their  description  of  MOEs  is  somewhat  general  and  does  not  really  provide  a  usable 
definition.  They  instead  developed  the  concept  of  “constants”  to  define  system 
performance.  Goode  and  Machol  [9]  provide  the  first  actual  definition  that  this  author 
could  find.  Their  definition  is  simple  and  straightforward: 

“  The  measure  of  effectiveness  is  the  criterion  by  which  solutions  will  be  judged  - 
proposed  solutions,  solutions  under  test,  or  solutions  in  being”. 

Goode  and  Machol  also  developed  a  set  of  desired  characteristics  for  MOEs  and 
recognized  the  possible  existence  of  multiple  measures.  A  1964  report  by  the  Weapon 
System  Industry  Advisory  Committee  (WSEIAC)  also  addressed  the  problem  of  systems 
effectiveness  [30].  This  report  gave  MOEs  a  systems  flavor  in  its  definition: 

“Systems  effectiveness  can  be  defined  as  a  measure  of  the  extent  to  which  a  system  maybe 
expected  to  achieve  a  set  of  specific  mission  requirements” . 
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Rudwick  further  notes  that  the  WSEIAC  approach  handles  the  measurement  of  any 
system  in  a  hierarchy  of  systems.  He  also  pointed  out  that  by  their  definition  of  system 
effectiveness,  the  result  is  always  measured  in  a  probabilistic  manner.  Ackoff  [1-3]  also 
arrived  at  the  same  conclusion,  i.e.,  a  system  of  systems  with  performance  results 
measured  probabilistically. 

Others  have  also  addressed  the  issue  of  MOEs  in  the  intervening  years,  but  there  are  two 
efforts  that  are  noteworthy.  First  is  the  MORS  work  that  will  be  described  shortly  [36- 
41].  Second  is  the  work  of  Noel  Sproles  [32-34]  who  has  written  extensively  on  the 
operational  definition  of  MOEs.  All  the  aforementioned  papers  will  provide  a  foundation 
for  a  theory  of  MOEs. 

Building  on  this  foundation,  the  goal  of  this  paper  is  to  present  a  theoretical  paradigm  for 
MOEs  that  is  grounded  in  the  first  principles  of  systems  theory.  The  result  will  be  a 
consistent  approach  that  provides  a  basis  for  developing  testable  MOEs  within  a 
mathematical  framework  that  allows  for  evaluating  the  statistical  significance  of  change 
when  performing  analysis  of  alternatives. 

The  Military  Operations  Research  Society  and  MOEs 

The  discussion  of  MOEs  in  the  literature  was  not  widespread.1  As  a  result,  prior  to  the 
Military  Operations  Research  Society  (MORS)  series  of  Command  and  Control  (C2) 
workshops  there  was  a  lack  of  consensus  among  analysts  and  decision-makers  over  the 
use  of  a  consistent  set  of  definitions  relevant  to  both  C2  systems  and  the  measurement  of 
their  performance.2 

The  MORS  study  charter  came  from  the  Deputy  Chief  of  Staff,  Plans  and  Programs, 
Headquarters,  Air  Force  Systems  Command,  who  challenged  Air  Force  planners  to 
evaluate  the  effectiveness  of  C2  systems.  He  noted  that  there  was  a  lack  of  analytic 
definitions  as  to  what  C2  architecture  is.  He  felt  that  there  is  a  need  as  well  as  a 
requirement  for  generic  tools  to  evaluate  C2  systems  and  architectures.  He  also  identified 
the  all  too  common  problem  that  such  tools  as  do  exist  are  usually  focused  upon  the 
specific  aspects  that  the  analyst  doing  the  problem  is  most  familiar  with,  regardless  of 
their  fit  to  the  problem. 

The  crux  of  the  problem  that  faced  the  first  workshop  participants  was  twofold.  First, 
how  to  establish  the  relationship  between  a  C2  process  and  the  physical  entities  that  are  a 
part  of  the  C2  system.  Second  was  how  to  evaluate  the  resulting  system.  To  accomplish 
this  they  had  to  develop  the  specification  of  an  appropriate  model  of  C2  and  the 
integration  of  the  selected  C2  model,  measures,  methods,  and  mathematics.  This 
involved  developing  an  approach  that  accounted  for  the  relationship  between  the  decision 
to  be  supported  and  the  analysis  itself  and  the  way  in  which  a  C2  model  may  be 
integrated  into  a  specific  problem. 


1  the  majority  of  the  known  sources  are  listed  in  the  Reference  section. 

2  C2  will  be  used  throughout  this  paper  in  a  broad  generic  sense;  i.e.,  the  concepts  apply  to  command,  control,  and  communications 
(C3),  command,  control,  communications,  and  intelligence  (C3I)  etc. 
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MORS  Workshops 

The  initial  Command,  Control,  Communications,  and  Intelligence  (C3I)  Measures  of 
Effectiveness  workshop  took  place  in  early  1984.  The  workshop  focus  was  on  the 
following  issues: 

•  C3I  model 

•  Transfer  functions 

•  Analysis  objectives 

•  Standard  terminology 

•  Methodology 

•  Measures  of  Effectiveness  applications 

•  Decision  making  and  decision  makers 

The  results  of  this  workshop  were  presented  at  the  52nd  MORS  symposium.  The  interest 
and  obvious  need  led  to  a  MORS  sponsored  workshop  in  January  1985  where  a  strawman 
C2  MOE  approach  was  developed  and  evaluated  [36].  The  results  of  the  work  with  the 
strawman  concepts  were  presented  to  a  special  session  at  the  53rd  MORS  symposium  and 
were  eventually  published  by  MORS  in  June  1986  [37]. 

In  January  1986,  the  workshop  participants  gathered  again  to  test  the  architecture  concept 
now  known  as  the  Modular  Command  and  Control  Evaluation  Structure  (MCES)  [39] 
that  had  evolved  from  the  earlier  work.  This  workshop  focused  on  four  applications  as  a 
test  of  the  MCES.  The  four  applications:  Navy  Battle  Force  Architectural  Study; 
Identification  Friend,  Foe,  or  Neutral  Test  Bed  Architectural  Study;  Strategic  Air 
Command  Operational  Testing  Study;  and  Global  Scale  Warfare  C2  Architecture 
(SuperCINC)  Study  demonstrated  that  the  MCES  approach  was  effective  over  a  range  of 
diverse  problems.  The  next  several  years  saw  the  approach  gain  acceptance  within  the 
Department  of  Defense  as  well  as  within  the  C2  community. 

In  the  fall  of  1992,  MORS  convened  another  workshop  to  “examine  the  context, 
processes,  and  methodologies  for  developing  and  using  C3IEW  [28] .”  The  impetus  for 
this  workshop  was  the  perception  that  the  rapid  growth  in  computer  and  communications 
technologies  required  that  the  previous  work  be  updated.  The  objectives  of  this 
workshop  were  similar  to  the  original  workshops;  i.e.,  it  was  to  assess  the  ability  of 
current  methodologies  to  evaluate  the  impact  of  C3IEW  on  campaign  force-level 
effectiveness.  It  was  to  identify  any  deficiencies  with  methods  and  tools  and  identify 
resolutions  to  these  deficiencies.  Again,  like  the  original  workshops,  this  workshop  led  to 
another  workshop  in  the  spring  of  1993.  The  purpose  of  this  workshop  was  to  evaluate 
the  results  of  the  first  workshop  using  the  example  of  the  unmanned  aerial  vehicle  (UAV) 
[35].  The  outcome  of  these  workshops  confirmed  the  validity  of  the  original  work. 

A  major  issue  is  that  the  results  appear  to  have  been  confined  to  the  C2  and  MORS 
community  even  though  several  books  have  published  [4],  [15],  [16],  [44]  that 
incorporate  the  approach..  Since  this  effort  there  has  been  little  to  no  work  in  this  area 


Electronic  Warfare  was  added  as  a  new  dimension  to  the  C3I  process  thus  C3IEW. 
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and  the  body  of  work  has  not  been  cited  in  any  of  the  eight  to  ten  MOE  focused  papers 
that  the  author  has  reviewed  in  the  last  ten  years. 

Review  of  the  MORS  Approach 

The  MORS  approach  consists  of  two  parts:  (1)  theory,  and  (2)  an  analytic  framework 
(MCES).  MCES  ties  together  a  number  of  conceptual  points  that  help  the  analyst  and  the 
decision-maker  to  better  understand  the  bounds  of  their  analysis. 

The  Theory 

The  MORS  approach  starts  with  a  set  of  standard  terminology  and  ideas.  The  goal  was  to 
provide  a  common  reference  point  in  order  to  promote  understanding  and  reduce 
controversy.  The  working  group  developed  the  following  definitions  as  their  starting 
point: 

•  C2:  “The  exercise  of  authority  and  direction  by  a  properly  designated  commander 
over  assigned  forces  in  the  accomplishment  of  his  mission.”  This  definition  was 
extracted  directly  from  the  Joint  Chiefs  of  Staff  Publication  1  (JCS  Pub  1). 

•  C2  system:  Viewed  as  having  three  components:  physical  entities,  structure,  and 
a  C2  process.  Physical  entities  refer  to  hardware,  software,  and  people.  Structure 
refers  to  the  relationship  between  physical  entities,  procedures,  protocols,  and 
concepts  of  operation  and  information  patterns.  It  can  reflect  the  effects  of 
doctrine,  the  scenario,  and  time  and  space. 

•  C2  Process:  The  C2  process  reflects  the  functions  carried  out  by  the  C2  system  - 
sensing,  assessing,  generating,  selecting  alternatives,  planning,  and  directing. 
Figure  1  represents  the  C2  process  interacting  with  its  environment  (This  model  is 
based  on  the  work  of  Lawson  [14]). 

•  Boundaries:  The  boundary  of  a  C2  system  is  a  function  of  the  system  under 
analysis  and  delineates  the  system  being  studied  from  the  environment.  The 
definitions  that  follow  apply  in  the  context  of  the  system  boundary  including  the 
environment. 

•  Dimensional  Parameters:  Properties  or  characteristics  in  the  physical  entities 
whose  values  determine  system  behavior  and  the  structure  under  consideration 
even  when  at  rest. 

•  Measures  of  Performance  (MOP):  Related  to  inherent  parameters  (physical  and 
structural)  but  measure  attributes  of  system  behavior. 

•  Measures  of  Effectiveness  (MOE):  Measure  of  how  the  C2  system  performs  its 
functions  within  an  operational  environment. 

•  Measures  of  Force  Effectiveness  (MOFE):  A  measure  of  how  the  C2  system, 
and  the  force  of  which  it  is  a  part,  performs  its  missions. 

•  Measures  of  Merit  (MoM):  A  term  defined  by  the  1992  workshop.  MoMs 
subsume  all  the  measures  that  characterize  a  C2  system.  The  context  in  which 
MoMs  are  measured  affects  the  way  in  which  they  are  defined.  Depending  upon 
the  analytic  perspective  a  MoM  could  be  a  MOP  or  a  MOE.  It  depends  upon  the 
question  being  answered  in  the  analysis.  MoMs  are  generally  not  multi-purpose; 
i.e.,  measures  that  are  used  to  evaluate  a  system’s  life  cycle  may  not  be  applicable 
to  measuring  operational  goals  (e.g.  doctrine  development).  It  is  highly  unlikely 
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that  a  single  set  of  measures  can  be  used  in  every  application,  or  that  each 
measure  selected  can  be  used  in  every  application. 


C2  Process  Model 

Lawson’s  C2  Process  Model 
Figure  1 


Figure  2  relates  the  C2  system  boundaries  to  the  various  levels  of  measures.  Dimensional 
parameters  and  MOPs  are  specified  and  measured  within  the  C2  system  boundary 
whereas  MOEs  and  MOFEs  are  specified  and  measured  external  to  the  boundary. 


C2  System  Boundary  Levels 
Figure  2 
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The  MCES  Process 

While  the  definitions  were  developed  for  evaluating  C2  systems  the  definitions  apply  to 
all  systems  and  systems  of  systems  as  well.  The  key  point  is  describing  the  system 
boundary  with  the  external  environment  and  the  processes  of  functions  that  are  contained 
within  the  boundary.  Figure  3  shows  that  the  first  steps  are  to  bound  the  system  and 
define  and  integrate  the  processes  of  interest;  only  then  can  the  measures  be  specified.  In 
systems  engineering  this  boundary  is  shown  in  a  context  diagram  such  as  that  shown  in 
Figure  4. 


The  MCES  Process 
Figure3 


C2  System  Context  Diagram 
Figure  4 
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Table  1  is  a  summary  of  the  characteristics  that  a  MoM4  should  possess  as  defined  by  the 
workshop.  This  list  is  similar  to  the  attributes  listed  by  Goode  and  Machol.  They  include 
the  requirement  that  the  MoM  be  efficient  in  the  statistical  sense  (small 
variance/reasonable  accuracy).  However,  neither  set  supports  the  concept  that  all 
measures  are  expressed  as  a  probability  as  advanced  by  Rudwick.  This  is  a  crucial  issue 
and  will  be  one  of  the  focal  points  for  the  rest  of  this  paper.  In  the  author’s  opinion,  the 
main  problems  that  most  analysts  have  with  development  of  MoMs  is  the  failure  to 
adhere  to  the  MCES  process  and  the  failure  to  realize  that  MoMs  are  related  to  the 
outcome  of  a  process.  To  this  extent  the  MORS  work  is  incomplete. 

Developing  the  Theory 

It  should  be  clear  that  the  MORS  work  represents  a  good  starting  point.  It  is  based  on  the 
knowledge  and  experience  of  many  people.  Thus,  while  several  definitions  for  MOEs 
have  been  advanced  in  recent  years  including  those  of  Sproles  and  Hockberger  [13],  this 
paper  will  argue  that  the  MORS  definitions  will  serve  well.  In  summary,  the  points  to 
build  on  are: 

•  The  importance  of  system  bounding; 

•  The  hierarchal  relationship  between  measures,  and; 

•  The  focus  on  process  and  resulting  interactions  with  the  environment. 

All  that  remains  is  to  place  them  in  a  true  systems  context  with  their  associated 
mathematical  relationship. 


Characteristics 

Definition 

• 

Mission  oriented 

• 

Relates  to  force/system. 

• 

Discriminatory 

• 

Identifies  real  difference  between  alternatives. 

• 

Measurable 

• 

Can  be  computed  or  estimated. 

• 

Quantitative 

• 

Can  be  assigned  numbers  or  ranked. 

• 

Realistic 

• 

Relates  realistically  to  the  C2  system  and  associated 
uncertainties. 

• 

Objective 

• 

Defined  or  derived,  independent  of  subjective  opinion 
(it  is  recognized  that  some  measures  cannot  be 
objectively  defined). 

• 

Appropriate 

• 

Relates  to  acceptable  standards  and  analysis 
objectives. 

• 

Sensitive 

• 

Reflects  changes  in  system  variables. 

• 

Inclusive 

• 

Reflects  those  standards  required  by  the  analysis 
objectives. 

• 

Independent 

• 

Mutually  exclusive  with  respect  to  other  measures. 

• 

Simple 

• 

Easily  understood  by  the  user. 

Desired  Characteristics  for  Measures  of  Merit 
Table  1 


Some  Basic  Systems  Theory 

Maier  and  Rechtin  [21]  define  a  system  to  be: 

A  set  of  different  elements  so  connected  as  to  perform  a  unique  function  not  performed  by 
the  elements  alone. 
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Measures  of  Merit  will  be  used  throughout  the  rest  of  this  paper  to  be  consistent  with  the  MORS  definitions. 


Ackoff  [3]  notes  that  “the  function(s)  of  a  system  is  the  production  of  the  outcomes  that 
define  its  goals(s)  and  objective(s).”  To  that  end  complex  man  made  systems  (C2 
systems,  modem  automobiles,  etc.)  exhibit  the  following  characteristics: 

•  They  are  open  systems  -  they  interact  with  their  environment  (as  per  the  Lawson 
model). 

•  They  exhibit  closed  loop  behavior  -  they  respond  to  the  environment. 

•  They  operate  in  a  continuous  or  discrete  state  but  not  steady  state. 

•  Systems  are  hierarchal  in  nature  -  examination  of  the  elements  or  structure 
provides  a  more  detailed  view  of  its  static  characteristics. 

•  Its  functional  properties  are  highly  differentiated  and  capture  behavior  or  the 
dynamic  characteristics  as  it  interacts  in  its  environment. 

Describing  the  dynamic  behavior  and  its  outcome  is  the  crux  of  the  analyst’s  problem. 
Does  the  set  of  behaviors  provide  the  expected  outcome?  By  the  requirement  for  the 
MoM  to  be  quantitative  the  expected  outcome  is  expressed  as  an  expected  value  or 
probability  of  occurrence.  Rudwick  notes  that  to  evaluate  system  effectiveness  the 
system  must  be  placed  in  its  operational  environment  and  operated  in  accordance  with  the 
specified  environmental  conditions  established  in  the  analysis.  As  mentioned  earlier,  he 
states  that  by  this  definition,  system  effectiveness  is  always  measured  in  a  probabilistic 
fashion.  Ackoff  defines  this  idea  mathematically  [2], 

E  =  f(ci,ui ) 


Where: 

•  E=  a  measure  of  the  performance  of  the  object,  organism,  or  organization 
involved. 

•  c,  =  the  set  of  controlled  variables. 

•  Ui  =  the  set  of  uncontrolled  variables. 

•  /=  the  relationship  between  the  preceding  variables. 

Referring  back  to  the  MORS  definitions  c;  and  u;  represent  the  parameter  set  of  the 
system  and  the  environment  respectively.  Ackoff  further  specifies  that  A  ( 1  <  /  <  m ) 

represents  different  actions  available  to  a  system  in  a  specific  environment  [3]  (a  change 
in  the  parameter  set  will  change  the  behavior).  P,  is  the  probability  that  the  system  will 
select  these  courses  of  action  in  that  environment.  Then: 

£p,= i.o 

i=i 

If  Ejj  represents  the  probability  that  a  course  of  action  A;  will  produce  an  outcome  0,  then 
the  efficiency  of  the  system  in  producing  the  outcome  Oj  is: 


p  =  y  PE 

o  l  1J 
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More  on  Processes 

As  noted  earlier  systems  will  instantiate  their  behaviors  either  continuously  or  discretely 
or  in  a  combination.  As  an  example  radar  can  search  (continuous)  and  detect  (discrete)  at 
the  same  time.  Processes  can  also  occur  sequentially  or  parallel  or  in  combinations 
thereof.  Thus  radar  will  search,  detect,  and  track  sequentially  but  can  also  search  and 
track  in  parallel  (once  the  target  meets  tracking  criteria). 


Start 


Serial  Processes 
Figure  5 

The  product  of  the  individual  outcomes  of  A,  B,  and  C  gives  the  overall  outcome  of  these 
processes. 


p  =  p  p  p 

1 1  1  A1  B1  C 


Outcome 


Parallel  Processes 
Figure  6 

For  a  parallel  network  the  overall  outcome  is  given  by 

p  =  p  +p  -P  p 

1 1  1  A~  1  B  1  A1  B 


Start 


Series-Parallel  Processes 
Figure  7 

For  a  series-parallel  network  the  overall  outcome  is  given  by 
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p  =  P(P  +p  -P  p  ) 

1  t  1  fV  A  ~  1  B  1  A1  B> 

Evaluation  of  more  complex  processes  can  be  accomplished  by  applying  the  mathematics 
of  reliability  theory  to  the  network  of  processes. 

What  Really  Constitutes  a  MOM? 

Rudwick  believed  that  systems  effectiveness  was  a  function  of  three  primary 
components: 

•  Availability, 

•  Dependability  (Reliability),  and 

•  Capability. 

Marshall  [23]  believed  that  system  effectiveness  or  probability  of  performance  (R/,)had  a 
fourth  component  -  survivability.  He  expressed  the  relationship  between  these 
components  as  follows: 


p  o  m 

Where  A  is  availability,  R  is  reliability,  S  C  is  capability  and  all  are 

What  both  have  said  is,  that  from  a  Systems  Theory  perspective,  the  dynamics  of  the 
system  interacting  with  its  environment  is  a  complex  process  that  involves  consideration 


An  Example 

Consider  the  performance  of  a  battlegroup  in  defending  itself  against  hostile  air  threats 

processes  for  this  system  of  systems  are  detection,  command  and  control,  and 
engagement.  Using  Marshall’s  approach,  the  efficiency  of  the  battlegroup  in  the  defense 

availability  of  AAW  assets,  reliability  of  AAW  assets,  Survivability  of  the  AAW  assets 
and  the  overall  engagement  capability  of  the  AAW  assets:  the  probability  of  detecting  the 

of  completing  the  target  weapon  interaction  successfully. 

A  detailed  breakdown  of  each  term  is  beyond  the  scope  of  this  paper.  However, 
the  probability  of  a  successful  engagement  P  is  given  by: 


se  d 


c  e 


searching  at  the  same  time)  so  the  performance  of  the  poorest  unit  could  drag  overall 
detection  performance  down. 
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The  Role  of  Time 

Analysts  frequently  use  time  as  a  performance  measure-  erroneously!  Their  error  is  that 
they  lose  sight  of  the  fact  that  the  goal  is  to  evaluate  the  outcome  of  a  process.  Because 
processes  can  combine  serially  or  in  parallel  time  is  the  independent  variable  and  the 
outcome  is  expressed  as  an  expected  value  given  a  time  window  or  the  expected  value 
that  a  process  can  be  completed  within  a  time  window. 

Summary/ Conclusions 

The  MORS  workshops  of  the  80’ s  and  90’ s  provided  a  solid  foundation  for  developing  a 
needed  mathematical  approach  to  MoMs.  This  body  of  work  emphasizes  that  to  properly 
develop  MoMs  the  analyst  must  focus  on  bounding  the  system  properly  and  describing 
the  processes  to  the  appropriate  level  of  abstraction.  Understanding  that  the  focus  is  on 
process  provides  for  an  appropriate  mathematical  formulation  of  the  process  and  its 
outcomes.  The  outcome  of  a  process  is  an  expected  value  based  upon  system  parameters 
for  a  given  environment.  In  addition,  a  true  system  assessment  may  require  the 
evaluation  of  several  factors 

This  paper  is  a  work  in  progress.  Future  work  will  include  extending  the  mathematical 
formulation  through  application  to  selected  problems  in  both  the  military  and  business 
domains. 
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